Proactive detection of malicious software, is a constant battle between anti-malware improvements and malware improvements, with each reacting to changes in the tactics and capabilities of the other, with attackers constantly evolving and evading existing defenses. Many approaches have been attempted in response to the problem of malware, each of which have positive and negative aspects. Some of the approaches previously tried include file inspection, file reputation checking, behavioral monitoring, whiteli sting, host-based intrusion prevention systems, network-based intrusion prevention systems, access protection, application control, sandboxing, etc. But malware continues to find ways to evade all of those techniques.
For example, the trustworthiness of a process/application can be determined based on many factors such as threat information of the binary, command line parameters, code signature or certificate, trustworthiness of the parent process etc. But what if a vulnerability of a trustworthy application is exploited to run a malicious payload? For example, one known malware example uses vulnerabilities in Microsoft's Internet Explorer® browser to rewrite code thereby making the browser execute a malicious payload. (INTERNET EXPLORER is a registered trademark of Microsoft Corporation.)
Polymorphic malware is another class of malware that changes its signature to evade detection by anti-malware programs. Packed malware is a type of malware that has been modified using a compression and/or encryption program. This compressed payload is appended or prepended to an executable containing the code to decompress it during runtime. This poses challenges to scanning engines as the malicious payload is encrypted.
For re-evaluating trust of processes, there are not many effective methods that are in practice. Most of the current approaches employ launch time evaluation without any effective runtime re-evaluations. There are a few solutions that deal with packers by emulating or detonating the binary in sandboxed environment. But advanced packers employ techniques like time bombs, polymorphism, environment awareness, etc. to render such techniques ineffective. Other techniques require reverse engineering and understanding how different packers work, which involves manual research and does not work for new packers or custom packers.
Improved techniques for detecting malware would be desirable.